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Abstract  -  The  gains  in  surveillance  information  that 
can  be  provided  by  fusion  of  multiple  sensors  have  been 
demonstrated  in  theoretical  and  practical  terms . 
However,  the  use  of  additional  sensors  quickly  reaches  a 
point  where  the  marginal  benefits  outweigh  the 
marginal  costs.  In  part,  this  is  due  to  an  increasing 
probability  of  misassociation.  Additionally,  the 
probability  of  finding  an  available  sensor  with  new 
information  decreases  as  a  wider  net  is  cast  around  the 
problem.  What  is  required  is  the  right  sensor  in  the 
right  place  at  the  right  time.  Clearly,  the  right  place  and 
time  cannot  always  be  known  in  advance  of  an  act  of 
hostile  intent.  The  hypothesis  is  that  the  use  of  mobile 
sensors  in  a  cued  mode  of  operation  can  provide 
quantitative  benefits  to  a  multi-sensor  surveillance  web 
employing  fusion  techniques.  The  mobility 
requirements  and  on-board  sensor  requirements  can  be 
studied  in  terms  of  the  characteristics  of  the  fixed  sensor 
web. 
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1  Introduction 

This  paper  is  organized  according  to  the  following  format. 
Section  2  builds  a  general  discussion  of  fusion  from 
homogeneous  sensors  to  non-homogeneous  sensors  to 
cued  fixed  sensors,  and  finally  cued  mobile  sensors. 
Section  3  tracks  the  growth  of  a  particular  data  fusion 
product  (GDFS  or  Graphical  Data  Fusion  System)  through 
expanding  requirements  and  applications  to  new  and 
evolving  situations.  Section  4  previews  present  and  near- 
term  future  work  at  NURC  in  this  area. 

2  General  discussion 

The  use  of  fusion  processing  in  cases  of  multiple  sensors 
which  are  homogeneous  or  nearly  homogeneous  has  been 
practically  applied  for  more  than  a  decade.  Homogeneous 
in  this  context  refers  to  sensors  that  exploit  the  same 
physical  phenomenon  (e.g.  electro-magnetic  radiation)  at 
about  the  same  frequency  (e.g.  X-band),  and  thus  have 
similar  specifications  in  terms  of  effective  range, 
resolution  of  localization  in  range  and  bearing,  update 
rates,  and  other  parameters  important  for  localization  and 
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tracking.  Prevalent  examples  include  multiple  fixed  radar 
sites  feeding  a  single  display  of  air  or  ship  traffic.  The 
motivation  for  fusion  of  such  data  is  one  or  more  of  the 
following: 

•Continuous  tracking,  areas  of  non-overlapping  coverage 
•Filling  coverage  gaps  in  presence  of  fixed  obstacles 
•Reduction  of  false  alarms,  areas  of  overlapping  coverage 
•Increased  probability  of  detection,  overlapping  coverage 
•Accurate  localization,  areas  of  overlapping  coverage 

The  use  of  non-homogeneous  (or  heterogeneous)  sensors 
falls  into  one  of  a  few  categories. 

•Complementary  overlapping  sensors  to  detect  &  localize 
•Complementary  non-overlapping  sensors 
•Cued  sensors  (fixed)  for  identification  and  assessment 
•Cued  sensors  (mobile)  for  identification  and  assessment 

The  term  complementary  is  used  here  to  indicate  that 
there  is  some  universal  set  of  all  information  available 
from  two  or  more  sensors.  Sensor  A  is  complementary  to 
sensor  B  if  at  least  a  subset  of  the  information  available 
from  A  is  disjoint  from  all  information  from  B.  The  key 
concept  here  is  the  fusion  of  independent  information. 

Complementary  overlapping  sensors  would  be  used  in 
cases  where  another  sensor  could  add  value  to  a  fused 
depiction  of  a  situation.  As  in  the  case  of  homogeneous 
sensors,  this  value  might  take  the  form  of  increased 
probability  of  detection  or  reduced  false  alarm,  or  other 
amplifying  data  to  the  situational  awareness.  The 
complementary  sensors  might  be  optimized  for  different 
environmental  conditions,  or  for  a  different  subset  of 
contacts  or  phenomena  being  observed.  The  use  of 
Doppler  radar  over  satellite  weather  imagery  is  an 
example  of  this  type  of  fusion. 

Complementary  non-overlapping  sensors  would  include 
the  fusion  of  radar  and  sonar  used  at  sea.  The  radar  can  be 
expected  to  display  information  about  surface  and  air 
contacts,  while  the  sonar  brings  information  on  subsurface 
and  surface  contacts.  The  surface  contacts  are  in  an 
intersection  set,  and  in  this  case  the  area  of  coverage 
overlaps  when  projected  to  a  planar  (map)  view. 
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Cued  sensors  generally  differ  from  detection  sensors  in 
their  field  of  view,  effective  range  and  available 
resolution.  An  example  of  a  cued  sensor  from  a  fixed 
location  is  the  use  of  a  camera  (visible  or  infrared)  trained 
to  the  location  of  a  radar  contact.  Cued  sensors  are  more 
likely  to  require  operator  interpretation,  but  with  that 
human-in-the-loop  assessment  comes  additional 
information  about  the  contact.  Note,  however,  that  a 
higher-resolution  cued  sensor  might  not  in  fact  have  a 
smaller  field  of  view.  Consider  the  case  of  a  surveillance 
radar  cuing  a  space-borne  or  airborne  synthetic  aperture 
radar  as  an  example  with  a  different  twist. 

One  might  consider  whether  it  would  be  better  to  hilly 
populate  (ensure  100%  coverage)  of  a  certain  area  by 
identification  sensors  instead  of  cuing  them.  The  three 
disadvantages  to  such  an  approach  are 

1.  Cost  of  additional  sensors  (cost  referring  not  just  to 
monetary  value,  but  impact  on  the  decision  process  as  a 
whole) 

2.  Requirement  for  additional  bandwidth 

3.  Additional  demands  on  fusion  processing. 

In  the  case  of  a  camera  used  for  identification  of  a  ship  or 
other  radar  contact,  it  is  clearly  more  cost  effective  to 
design  a  slew  and  assess  mechanism  than  to  span  a 
significant  sector  of  coverage  with  multiple  sensors,  each 
with  a  narrow  field  of  view  and  possibly  a  small  range  of 
focus.  Using  the  term  “focus”  broadly,  we  are  interested 
in  focusing  the  attention  of  the  identification  sensor  on  the 
location  of  a  contact  detected  by  another  sensor.  In  fact, 
this  suggests  a  more  fundamental  reason  that  numerous 
identification  sensors  might  not  be  appropriate  for  filling 
the  observation  space.  If  that  were  done,  the  size  of  the 
team  of  operators  would  grow  without  bound,  or  a 
distillation  process  must  be  added  to  automatically  bring 
events  of  significance  to  the  operator. 

By  using  cued  sensors  instead  of  an  unrealistic  net  of 
high-resolution  sensors,  there  is  an  implication  of  resource 
management.  The  duality  between  data  fusion  and 
resource  management  is  presented  in  Steinberg  and 
Bowman’s  2004  work  [ref].  In  this  case,  it  implies  at  least 
cursory  activity  at  higher  levels  of  fusion  (situation 
assessment,  impact  assessment,  and  process  refinement) 
before  cuing  a  limited  resource  to  continue  the  object 
assessment  level  from  “location  and  track”  to 
“identification  and  classification.”  It  is  important  to  note 
that  this  involves  an  escalation  of  the  mission,  and  an 
overt  resource  management  decision  -  whether  made  by 
human  or  automatic  means.  The  important  inter-process 
communication  which  must  be  established  is  timely  and 
accurate  updates  of  the  location  of  the  contact.  Since 
contacts  in  motion  are  of  interest  (in  fact,  one  might  say 
that  interest  is  proportional  to  speed)  the  location  data 
should  include  predictions  of  motion  so  that  the  cued 
sensor  can  smoothly  follow  the  contact  in  motion  from 
one  position  update  to  the  next. 


The  use  of  a  specialized  sensor  for  identification  or  other 
fine  augmentation  the  data  fusion  picture  opens  the 
possibility  of  using  a  sensor  with  a  much  shorter  effective 
range  than  the  primary  detection  and  surveillance  sensors. 

It  was  proposed  that  the  fixed  identification  sensor 
considered  in  the  previous  paragraph  would  benefit  from 
rotating  as  a  cost-effective  alternative  to  populating  a  full 
circle  of  coverage  with  individual  cameras.  The  logical 
extension  of  that  situation  to  one  where  a  sensor  has  a 
limited  range  as  well  as  a  limited  field  of  view  is  that  the 
sensor  must  have  the  ability  to  translate  through  the  area 
of  interest.  There  are  a  wide  variety  of  vehicles,  manned 
and  unmanned,  which  may  prove  applicable  to  such  a 
mission.  The  spectrum  of  aircraft,  land  vehicles,  surface 
vessels,  or  submersibles  could  be  considered  in  general 
terms.  The  requirement  for  a  timely  feedback  mechanism 
becomes  more  critical  than  it  was  in  the  case  of  a  fixed 
sensor  rotating  to  a  particular  field  of  view.  In  the  case  of 
a  truly  mobile  sensor,  the  intercept  between  the  moving 
contact  and  the  mobile  sensor  must  continuously  be 
updated.  Based  on  those  updates,  the  vehicle’s  speed  and 
course  must  be  controlled,  whether  automatically, 
remotely,  or  by  direct  manual  means.  Additionally, 
aiming  of  the  sensor  may  be  required.  At  some  point  in 
the  approach,  the  contact  of  interest  will  be  re-acquired  by 
the  mobile  sensor,  which  is  likely  to  make  the  final  stage 
positioning  for  identification  and  observation  easier. 

3  A  coastal  and  harbour  surveillance 
Application 

The  US  Navy’s  Graphical  Data  Fusion  System  (hereafter 
GDFS)  was  first  released  in  1995  for  use  by  the  Mobile 
Inshore  Undersea  Warfare  Units.  These  commands  have 
since  been  reorganized  as  the  Navy’s  Mobile 
Expeditionary  Security  Force,  Sensor  Detachments.  The 
software  was  developed  as  a  joint  effort  by  a  team 
comprised  of  CEA  Pty.  Ltd.  of  Canberra,  Australia,  SAIC 
of  San  Diego,  CA,  and  SPAWAR  Systems  Center  of  San 
Diego,  CA.  Similar  and  spinoff  projects  were  also 
developed  for  the  US  Department  of  Energy,  Sydney 
Harbour  Port  Operations,  the  Bahrain  Coast  Guard,  and 
others.  The  system  has  recently  been  used  on  an 
experimental  basis  by  NURC  in  La  Spezia  Italy. 

The  first  operational  builds  of  the  software  fused  tracks 
from  multiple  radars  in  a  given  operating  area,  and 
displayed  the  tracks  on  a  geographical  display.  The 
display  was  comprised  of  two  map  windows  which  could 
be  controlled  (zoom  and  pan)  separately.  PPI  (plan 
position  indicator)  video  from  one  or  more  overlapping 
radars  could  be  selected  for  overlay  on  the  track  display. 
Additional  information  appeared  as  text  fields,  action 
buttons,  and  status  indicators  on  other  parts  of  the  screen. 

The  system  was  fused  with  cameras  (visible  and  infrared) 
which  could  be  controlled  directly  from  the  GDFS  console 
to  automatically  slew  to  a  given  point  on  the  screen  or 
follow  a  given  track.  The  camera  video  was  displayed  on 
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separate  screens  adjacent  to  or  above  the  main  computer 
screen  or  screens.  An  example  of  the  GDFS  screen  is 
shown  in  figure  1 . 


Figure  1:  Sample  GDFS  screen  during  operation.  Note  the 
two  map  windows  which  dominate  the  screen  area,  along 
with  track  data  information  on  the  lower  left,  and  general 
system  information  in  the  lower  right. 

In  the  mid-1990s,  the  system  was  expanded  to  receive  and 
fuse  data  from  passive  acoustic  and  ESM  sensors.  These 
sensors  generally  provided  bearing-only  data  (at  best)  and 
were  often  of  limited  use  to  the  fused  data  picture  in 
cluttered  harbours  and  approaches. 

The  original  software  ran  on  two  networked  UNIX 
workstations.  Near  the  end  of  the  20th  century,  the 
software  was  converted  from  Unix-based  to  MS 
Windows.  As  required  computing  resources  continued  to 
shrink  with  advancing  technology,  slightly  trimmed  down 
versions  of  GDFS  have  been  developed  to  run  on  laptop 
or  tablet  PCs.  These  versions  include  all  the  core  fusion, 
tracking,  display,  and  sensor  control  functions  of  the 
original  software,  and  are  only  reduced  in  the  number  of 
sensors  they  can  access  at  one  time.  However,  the  new 
GDFS  Lite  can  be  networked  and  its  data  fed  to  a  meta¬ 
fusion  level  GDFS  which  now  runs  on  a  rack-mounted 
windows  PC  at  a  central  location. 

The  important  sensor  assets  that  have  been  added  to  the 
architecture  in  the  current  decade  are 

•  Automatic  Identification  System  (AIS) 

•  Monostatic  Active  Sonar 

•  Response  vehicle  vectoring 

•  Response  vehicle  sensor  data 

AIS  is  a  system  which  digitally  transmits  vessel 
information  including  name,  location,  course  and  speed 
via  a  VHF  radio  link.  It  is  required  on  all  vessels  larger 
than  300  gross  tons,  and  is  used  by  port  operations  and 
vessel  tracking  systems  worldwide.  The  primary  purpose 


of  AIS  is  collision  avoidance,  but  with  appropriate  fusion, 
this  provides  an  easy  way  to  identify  many  radar  contacts 
from  large  vessels  in  a  crowded  port  environment.  Since 
the  update  rate  is  quite  slow,  it  is  of  limited  utility  in  close 
quarters  when  not  fused  with  radar  data. 

The  monostatic  active  sonar  interface  is  geared  towards  an 
underwater  threat  close  to  a  protected  asset.  The 
particular  sonars  which  have  been  interfaced  to  GDFS  are 
designed  primarily  for  detection  of  divers  or  swimmers, 
and  other  moving  contacts  of  similar  size  and  speed. 

The  interface  to  response  vehicles  or  vessels  is  a  very 
general  interface  which  allows  a  mobile  asset  with  some 
form  of  identification  sensor  to  be  cued  to  a  particular 
location.  Often,  this  is  in  response  to  a  contact  detected 
by  one  of  the  other  sensors  -  radar  or  sonar  for  example. 
The  interface  allows  for  data  to  flow  back  from  the  mobile 
sensor  to  the  main  control  station,  allowing  an  operator  to 
assess  and  identify  from  a  single  location.  The  interface 
has  been  used  in  this  manner  with  unmanned  aerial 
vehicles  and  unmanned  surface  vehicles,  most  notably  in 
the  Counter-Maritime  Improvised  Explosive  Device 
(CMIED)  Limited  Military  Utility  Assessment  in  the  fall 
of  2006. [2] 


Figure  2:  Sample  BDU  screen  during  operation.  Note  the 
map  window  at  left,  and  the  own- ship-relative  vector 
display  at  upper  right.  The  lower  right  of  the  screen  can 
show  either  general  status  display  (similar  to  figure  1)  or 
identification  sensor  information. 

In  the  case  where  the  response  vessel  is  manned,  a 
simplified  version  of  GDFS  is  installed  on  the  vessel  to 
assist  the  crew  in  locating  the  contact  of  interest.  This 
capability  was  also  first  shown  in  2006.  See  figure  2  for 
an  example  of  the  Boat  Display  Unit  (BDU)  screen.  The 
upper  right  portion  of  the  screen  is  dedicated  to  a  vessel- 
relative  bow-up  steering  display,  which  provides  simple 
feedback  to  the  coxswain  to  assist  in  vectoring  to  the 
contact.  The  bow-up  ring  display  was  used  earlier  on  a 
US  Coast  Guard  system  conducting  a  similar  mission.  In 
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2007,  NURC  scientists  and  engineers  expanded  the  utility 
by  vectoring  an  unmanned  vehicle  to  an  unseen 
underwater  contact  which  was  detected  by  active  sonar. 
The  unmanned  vehicle  carried  a  short-range  detection 
sonar,  which  fed  imagery  back  to  the  shore  GDFS  station 
in  real  time  for  assessment  by  the  operator. 

In  summary,  the  architecture  and  design  GDFS  has  proven 
to  be  flexible  and  versatile  in  the  face  of  numerous  new 
requirements  and  adaptations  of  the  mission  it  supports. 

4  Conclusions  and  future  work 

The  value  of  any  new  sensor  to  a  fusion  picture  should  be 
considered  carefully  before  assuming  that  it  will  aid  the 
operator.  The  value  of  a  cued  surface  vessel  carrying  an 
identification  sensor  would  be  useless  without  the 
expectation  that  it  could  make  an  intercept  in  a  timely 
fashion.  Consider  a  detection  sensor  with  an  effective 
range  of  Ad.  Suppose  a  response  vessel  with  an 
identification  sensor  has  a  maximum  practical  speed  of 
Vr.  Suppose  the  sensor  it  carries  has  an  effective  range  of 
Ar  «  Ad.  In  effect,  this  forces  the  effective  radius  of  the 
mobile  sensor  to  be  driven  by  the  vehicle  velocity,  not  the 
range  of  the  sensor.  The  effective  radius  of  the  mobile 
identification  sensor  is  then  VrT  where  T  is  some 
reasonable  time  in  comparison  to  the  approach  speed  of  a 
potentially  hostile  contact.  In  fact,  if  the  maximum 
assumed  velocity  of  a  hostile  contact  is  Vh,  then  the  time 
before  it  reaches  the  detection  sensor  is  T=A/Vh.  Note 
that  the  detection  sensor  needn’t  be  co-located  with  the 
protected  asset,  but  for  a  first  approximation  assume  that  it 
is.  The  result  of  this  rough  calculation  is  that  the  effective 
range  of  the  identification  sensor  on  the  mobile  platform 
is 

V 

VrT  =  VrA/Vh  =  A  — 

Vh 

Further  detail  of  such  analysis  is  ongoing  as  part  of  the 
reporting  from  the  Response  Against  Diver  Intrusion 
(RADI)  trial,  which  was  held  in  La  Spezia  last 
November. [3]  A  related  effort  in  conjunction  with  the 
University  of  Pisa  modeled  the  ability  of  various  vessel 
configurations  to  patrol  and/or  identify  detected  contacts. 
The  assessment  of  that  model  and  its  conclusion  is 
occurring  concurrently. 

The  three  components  mentioned  above  (analysis  as  in  the 
previous  paragraph,  modeling  in  conjunction  with  the 
University  of  Pisa,  and  experimentation  performed  in  late 
2007)  show  preliminary  indications  that  there  are  some 
conditions  under  which  mobility  in  conjunction  with 
identification  sensors  can  benefit  a  general  fusion  system. 
Continued  efforts  will  further  formalize  the  measures  of 
effectiveness  for  mobility  in  a  system  of  fused  sensors  in 
terms  of  the  salient  characteristics  of  the  sensor  and  the 
vessel. 


Another  direction  of  future  efforts  involves  the  fusing  of 
detailed  sonar  imagery  from  multiple  sources.  This  could 
include  on-vehicle  fusion  of  side-looking  and  forward- 
looking  sensors,  or  it  could  involve  fusion  of  fixed  sonar 
data  (wide  field  of  view)  with  detailed  imagery  data  from 
a  mobile  sensor  (narrow  field  of  view,  and  better 
resolution  in  both  horizontal  dimensions).  An  example  of 
such  a  depiction  of  a  diver  intrusion  is  shown  in  figure  3. 
Note  that  this  is  a  significant  departure  from  current 
efforts  which  expect  only  a  very  low  bandwidth  data 
stream  from  the  detection  sonar  which  updates  the 
location  of  moving  contacts  on  each  ping  cycle. 


Figure  3  :  Example  of  use  of  wide  and  narrow  sector  sonar 
together 


It  is  expected  that  the  outcome  of  these  two 
complementary  efforts  in  2008-9  will  lead  to 
specifications  and  a  prototype  of  an  unmanned  surface 
vehicle  which  can  be  used  to  intercept,  observe,  and  warn 
underwater  contacts  that  they  have  entered  an  exclusion 
zone.  Furthermore,  the  use  of  a  surface  vessel  positions 
NURC  to  perform  additional  work  against  surface  threats 
in  2009. 
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